The present invention generally relates to capturing, processing, and using a plurality of multimedia data sets for electronic systems. More specifically, the invention relates to one or more systems and methods for synchronizing, merging, and utilizing multiple data sets for augmented reality applications. Furthermore, the invention also relates to identifying, time-synchronizing, and creating a merged graphical and/or audio information data set and a non-graphical and non-audio information metadata from a plurality of data sets for subsequent use in augmented reality applications. In addition, the invention also relates to replaying or creating an augmented reality environment capable of dynamically-selecting and dynamically-changing information associated with a particular file or a particular geographical location.
Visual monitoring systems, such as closed-circuit televisions (CCTV's) for buildings and strategic locations and vehicle black boxes, are widely used in today's consumer and business electronics market. Many of these visual monitoring systems are able to capture, process, and record visual information as well as aural information in a digitized format in a local storage or a network-attached storage. In some instances, different types of visual information from multiple cameras or multiple data sources are juxtaposed and presented via a user interface in a single display screen. Furthermore, the user interface is also typically able to select one particular aural information associated with a particular visual information among a plurality of visual information via speakers in the single display screen.
In conventional visual monitoring systems, different types of visual information are kept as separate file formats in a data capture mode or a recording mode. For example, a live recorded video through a front windshield of a vehicle is in an independent multimedia data format, such as MPEG2, MPEG4, and WMV. If there is another live recorded video (e.g. through a rear windshield, a vehicle's cabin, and etc.) associated with a conventional visual monitoring system, then even if the two videos are displayed via a same user interface on a single display screen, the two video files typically remain separate and independent from each other. The separation of data formats among multiple visual, audio, and/or sensory information sources is even more likely if a type of visual, audio, and/or sensory information source is fundamentally different (e.g. camera feed vs. GPS map information), because the file format types themselves may be entirely different.
In some instances, augmented reality applications that require multiple visual, audio, and/or sensory information sources may encounter technical and logistical challenges in recreating an augmented reality environment in a single user interface in near real-time, especially if wireless transmission methods are used, because the multiple visual, audio, and/or sensory information sources in separate and different data formats have to be transmitted, received, and then synchronized to ensure that the referenced time stamps in the separate and different data formats are properly corresponding to a same time reference frame. Even though conventional data buffering methods may alleviate some technical difficulties in accommodating the augmented reality environment that receives, processes, and synchronizes multiple visual, audio, and/or sensory information sources from remote locations, multiple visual, audio, and/or sensory information sources in separate and/or different data formats are more likely to suffer data streaming glitches due to data network congestions than a single data format in most augmented reality applications. Data download synchronization challenges may be significant for an augmented reality application, if multiple visual, audio, and/or sensory information sources in separate and/or different data formats are to be received by the augmented reality application at a remote location and then displayed on a display screen simultaneously.
Furthermore, if a user desires to recreate an augmented reality environment based on a multiple number of previously-recorded visual, audio, and/or sensory information sources by downloading a plurality of separate and different data files in multiple formats, a user interface and an associated augmented reality application may also encounter time synchronization and technical difficulties in displaying the multiple number of previously-recorded visual, audio, and/or sensory information simultaneously, because the referenced time stamps in each data format may not be correctly synchronized for the user interface and the associated augmented reality application.
For example, if an augmented reality application is designed to display a frontal view, a driver's side view, and a rear view of a user's vehicle, while also displaying GPS map data and a second-by-second location of the vehicle, conventional visual monitoring systems for augmented reality require three separate video files for three different angles of the vehicle, a map data-related file, and a GPS location log for the vehicle which can be superimposed on the map data-related file in a display screen. Recreating an augmented reality environment by streaming multiple sources of data files in a data transmission network (e.g. wired, wireless, and/or cellular networks) presents significant technical difficulties for the augmented reality application that receives and attempts to time-synchronize the multiple sources of data files for a simultaneous display of multiple visual information.
In addition, in some augmented reality applications, it may be advantageous to enable a user to make dynamic changes and selections in a replay video or in a real-time augmented reality environment for a particular geographic location, so that the user can dynamically change or select different viewing angles and different information from previously-recorded visual, audio, and/or sensory information for presentation in the replay video or in the real-time augmented reality environment for the particular geographic location.
Therefore, it may be desirable to devise a system and a method that can efficiently process and readily synchronize a plurality of visual information data in augmented reality applications. Furthermore, it may also be desirable to devise a system and a method which are configured to merge a plurality of visual information data while retaining a non-graphical and non-audio portion of information associated with the plurality of visual information data as separate metadata variables from graphical information. In addition, it may also be desirable to provide an augmented reality application with a capability to enable a user-triggered dynamic change or dynamic selection from previously-recorded visual, audio, and/or sensory information in a replay video or in a real-time augmented reality environment by retaining a non-graphical and non-audio portion of information as separate variables from graphical information.